Thouless- Anderson-Palmer equation and self-consistent signal-to-noise analysis 
for the Hopfield model with three-body interaction 
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The self-consistent signal-to-noise analysis (SCSNA) is an alternative to the replica method for 
deriving the set of order parameter equations for associative memory neural network models and is 
closely related with the Thouless- Anderson-Palmer equation (TAP) approach. In the recent paper 
by Shiino and Yamana the Onsager reaction term of the TAP equation has been found to be obtained 
from the SCSNA for Hopfield neural networks with 2-body interaction. We study the TAP equation 
for an associative memory stochastic analog neural network with 3-body interaction to investigate 
the structure of the Onsager reaction term, in connection with the term proportional to the output 
characteristic to the SCSNA. We report the SCSNA framework for analog networks with 3-body 
interactions as well as a novel recipe based on the cavity concept that involves two cavities and the 
hybrid use of the SCSNA to obtain the TAP equation. 

PACS numbers: 75.10.Nr, 84.35.-K, 87.18.Sn 



The replica method [jj for random spin systems has 
been successfully employed in neural network models of 
associative memory @, tj ■ However the replica calcula- 
tions require the concept of free energy. On the other 
hand an alternative approach to obtain the order param- 
eter equations of the self-consistent signal-to-noise anal- 
ysis (SCSNA) [J, 0| for deterministic analog neural net- 
works is free from the energy concept and thus applicable 
to networks with asymmetric connections. The SCSNA 
was shown to be closely related to the TAP equation ap- 
proach H B 0, S through the cavity concept in the 
case where systems have an energy function [ppf . 

An advantage of dealing with the TAP equation of neu- 
ral networks is that equilibrium behaviors of a stochastic 
neural network can be studied by investigating the corre- 
sponding TAP equation that is viewed as a deterministic 
analog network jfj . The set of order parameter equations 
of the original stochastic network is obtained by applying 
the SCSNA to the TAP equation. 

TAP equations, which have recently been attracting 
much attention from the viewpoint of applying statisti- 
cal mechanics to information theoretic engineering prob- 
lems 0, H2I. a re usually obtained by means of the Ple- 
fka method |13| . A more systematic method of deriving 
TAP equations incorporates the cavity concept [3, Hj to 
elucidate the structure as well as the meaning of TAP 
equations. Noting that the SCSNA and the TAP equa- 
tion approach share a common idea of the cavity concept, 
Shiino and Yamana have found that the Onsager reaction 
term characteristic to a TAP equation can be recovered 
by the SCSNA for stochastic analog networks with 2- 
body interactions |10| . The aim of the present paper is 
studying the SCSNA framework and the TAP equation 
of stochastic analog networks with multibody interaction 



to elucidate the structure of the Onsager reaction term 
in connection with the term proportional to the output 
characteristic to the SCSNA. 

The Ising spin Hopfield models with p-body connec- 
tions Il4| which are analogous to p-spin random spin glass 
models |l5| were studied to explore the statistical behav- 
ior of their retrieval properties. It is wcllknown that the 
storage capacity of the network is proportional to A p_1 
fl4| where N represents the number of neurons. In the 
present paper we report a novel method based on the 
cavity approach 2] to derive the TAP equations for the 
p = 3 Hopfield model with simultaneous use of the SC- 
SNA for this model. Since we deal with analog neurons, 
or soft spins in the present paper, we can choose trans- 
fer functions of arbitrary shapes. The TAP equations for 
analog networks with multibody interactions have not 
been reported. 

We deal with a stochastic analog neural network of the 
form 



dxi 
IF 



- ^ Ji]kX 3 X k + T}i(t) , (1) 



where Xi represents a state of an analog neuron or a 
soft spin at site i, 4>{ x i) the potential, rfo the Langevine 
white noise obeying (t]i(t)r]j (t 1 )) — ^S(t — t r )Sij for 

i = 1 • • • N, [3 the intensity of externally driven Langevin 
noise and the synaptic coupling Jy^ is assumed to be 
given by the Hebb learning rule extended to the p = 3 
Hopfield model 0: 



Hjk 



1 p 



(2) 



±1 
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for i 7^ j 7^ k and otherwise Jjyk = with tg 
representing p(— aN 2 ) random memory patterns. 

For the p — 3 Hopfield model, the local field at site 
i is defined as hi = J2j<k(yH) JijkXjXk and we have to 



2 



calculate the second moments of soft spins (xjX k ) to ob- 
tain the TAP equation which is given by expressing the 
thermal averages of the local fields in terms of those of 
soft spins. For this reason the standard cavity method 
applied to networks with p = 2, where it suffices to take 
only one cavity into account, is ineffective in the present 
case. To study the TAP equation for the system (|TJ we 
have to introduce two cavities. Hereafter we will refer to 
the sites i = 0, — 1 as the cavity sites. 

The Fokker-Planck equation for the probability den- 
sity of soft spins corresponding to the set of Langevin 
equations QJ with A — N + 2 has the equilib- 
rium probability density P eq (x-\, Xo,x±, ■ ■ ■ , Xn) = 
z ^ +2 exp (— @H( N+2 ') , where Zn+2 is the partition func- 
tion of the system including two cavities and H^ N+2 ^ is 
its Hamiltonian. Noting that the Hamiltonian of A-body 
system {x%, • ■ ■ ,xn} is given as 

N N 
H (N) _ ^2<j)(Xi) - E JijkXiXjXk , (3) 
i=l i<j<k=l 

the total Hamiltonian H^ N+2 ^ of the system which in- 
cludes the effects of cavity neurons is expressed as 



H ( N +V =HW+<f>(x ) + <f>(x-i) 
-h %o - h-ix-i - h-ioxox-i , 

where ho, h—i and /i_io are respectively defined as 



(4) 



1 1 N 



2 (A + 2) 2 



H j,k=l 
N 



i> - = iv^-^J2 E e^e k ^x k , (6) 



2 (A + 2) 2 



A» j,k=l 
3^k 



N 



V ' n fe=l 



Notice that ho and h-i are the local fields induced by 
the A neurons at cavity sites i — 0, — f respectively and 
h-xo is the local field of another type affecting the pair 
of cavities. Then using the marginal probability density 
and noting that the marginal probability density of the 
local fields of A-body system is expected to be a three- 
dimensional Gaussian distribution since ho, h-\ and h-\o 
are the sums of independent random variables, one can 
evaluate the (A + 2)-body average of the cavity soft spin 
xq as 



((ho)*) + (P(h- io)n + P 2(j2 -io 
x G((ho) N ) -F 2 ((h Q ) N )} fUK-Jn 



where F is the transfer function which is defined as 

f3 2 a 2 x 2 



F{y) = / dxx exp —/3<j>(x) + f3yx + 



P 2 a 2 x 2 



x < dx exp —f3(j)(x) + Pyx + 



(9) 



and 



G{y) = / dxx 2 exp 
x < / dxcxp —p<j>[x) + Pyx + 



P 2 a 2 x 2 



x) + Pyx + 
P 2 a 2 x 2 



(10) 



a 2 the variance of the local field hi, <J 2 _ W the covari- 
ance of ho and and (-)n the A-body average. Since 
(h_ w ) N = 0(1/ VN), a 2 _ w = 0(1/VN), 



(ho) N +2 = (h ) N + P<7 2 (x q )n+2 + O [l/VNj , (11) 

and the true local field ho is given by ho = ho + h—ioX—i, 
we have 

(x )n+2 = F ((h ) N+ 2 - P<J 2 (x ) N+2 ) ■ (12) 

This equation coincides with the one obtained using the 
standard cavity method for the Hopficld model where one 
cavity is taken. 

However, it does not suffice to obtain the TAP equation 
in the present case, because {ho} n+2 is simply given in 
terms of the second moments of soft spins (xjX k ) n+2- 
In order to find the expression of the local field (hi) n+2 
in terms of the thermal averages of soft spins, it will be 
necessary to employ 2-cavity method and we obtain 

(xqX-i)n+2 = (xo)n+2(x-i)n+2 



+ (/?(fc_io)jv+2+j9V 



in 



x ((^0)^+2 ~ (xq)n+2 2 ^ ( {x 2 _ 1 )n+2 - (£-i)at+2 2 ) 

+0(1/A). (13) 

Thus the thermal average of the local field (hi) n+2 is 
given as 



(hi) N+2 — E Jijk( x j) N+2(xk) N+2 
j<k&i) 

+^(x 1 ) n+2 u 2 + o{i/Vn) , 

where U is the average of susceptibilities 



U 



N + 2 



E (( x2 ])N+2 - (Xj)jV+2 2 ) 



(14) 



(15) 



+0(1/A) 



(8) 



Notice that (hy) n affects the (A+2)-body average of the 
local field (hi) n+2 via the summation with respect to site 
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index 

field. Eq.lE 
equations [T 



on the other hand <r 2 _ w does not affect the local 



together with Ea.lfT2)l yields the pre- TAP 
for p = 3 Hopfield model; 



{XijN = F | ^ Jijk{x 3 )N{Xk)N - r TA p(xj) w 

(*=!,•■• ,JV)(16) 



with 



and is assumed to be a Gaussian random variable 
with mean zero and the variance is to be evaluated self- 
consistently. 

Substituting the expression of the local field Ea. ll9|) 
into the pre- TAP equation (|16|l and comparing with 

(x/)n = F ^(/ij)jv-2j, which holds by observing Eq.lJSJ 
for large N, it follows that Tscsna = Ttap ,10], since 
(hi)N-2 — ^£,i( m1 ) 2 obeys a Gaussian random variable. 
Then we have 



TAP 



/3a 2 - 



aU 2 

2/3 ' 



(17) 



where (x{)n is used instead of (xi)N+2- — Txap (^i) a is 
the Onsager reaction term of the TAP equation. How- 
ever, Ttap, especially, a 2 in Eq.© has still to be deter- 
mined to obtain the TAP equation. 

Ea. i|16|) defines a deterministic analog network of as- 
sociative memory with the transfer function F corre- 
sponding to the original stochastic analog neural network 
0. To obtain a 2 together with the order parameter 
equations we then apply the SCSNA to Ea. ((TT))l . The 
SCSNA is a self-consistent method for properly renor- 
malizing the so-called noise part due to interference of 
noncondensed patterns in the local field of a neuron 
hf ct = J2j<k(^i) Ji]k(xj)N(xk)N in Eq.JTSl, which can 
be rewritten as 



= F (m 1 



from which it follows 

N 

X 



2 <n 



V ; N 



(24) 



in' 



i=l ' 

i=i v / 



c 

N 



+0 (V~ 5/2 ) 



(25) 



off' 



^ ct = \ E * 



{Xj/N 

N 



N 



i 



N 

tN' A , , 2 aN 



E 



N 2 



E#> ( 18 ) 



where m p is the overlap of // pattern defined as 
- J2iLi £f ( x i)n for M = 1, ' ' ' ! olN 2 . To extract the pure 
noise obeying a Gaussian distribution, we decompose the 
local field hf et assuming that the first pattern is con- 



where F' denotes the derivative of the transfer function 
F. 

It should be noted that in the present case it is neces- 
sary to obtain m fi (= 0(1/ y/N)) up to 0(1 /N) unlike 
the case of p — 2 where up to 0(1/ y/W) of m M suffices. 
We solve this equation for m/ 1 perturbatively by putting 
m M = rriy 2 + m 1 ^ + O (TV -3 / 2 ), where represents the 

part of 0(N~ k ). Substituting the solution of Eq.JSSJ into 
the definitions of Tscsna and z^, we find 7 = in the 
limit of large N. Furthermore, Tscsna which determines 
the form of the transfer function F by the relation 



SCSNA 



r 



TAP 



f3a 2 



all 2 



(26) 



densed and the others are noncondensed, i.e., m 1 = 0(1) is evaluated using the self-averaging property as 



and m» = 0(1/ y/N) (p > 2) as 



^ det =^K) 2 +^r 



(m^y 



G_ 

N 



(Xj}_N_ i 

N '' 



where U is given as 



+ 



\Xj)N' 
N 2 



£,i + z ifJi + Tscsna(xi)n ,(19) 



U 



where 



(28) 



N' 

i=l 

TSCSNA = 7^7; 



Ziu, + l\Xi) N 



i=l 

i — 7 , 

/j>2 
9 E 



7 



I v\2 C 
(TO ) 



(20) 

(21) 
(22) 

(23) 



and (-^i g represents the average over the random pat- 
tern £ = ±1 and the Gaussian random variable z. Notice 
that U given by this equation corresponds to that defined 
by Ea. (|15fl . since differentiating Eq.Q with respect to y 
yields the susceptibility and the self-averaging property 
holds. Similarly the variance of the Gaussian noise z can 
be calculated as 



(29) 
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Furthermore the overlap of the condensed pattern m 1 is 
given as 




Eqs.|2BJ), 123, GUI and (EDI) constitute the SCSNA 
framework in terms of m , U, a 1 , and er^ that yields 
the order parameter equations for determining the stor- 
age capacity of the analog network (|16|l and hence 
of the stochastic analog neural network Wc 
note that they coincide with those obtained by the 
replica calculation when the replica symmetry is as- 
sumed to hold. Furthermore, in the special case where 
exp -P4>{x)/ f exp -f3(p(x)dx = \ (6(x + 1) + S(x - 1)), 
the result of the SCSNA recovers that of the Ising spin 
network studied in plj . Then, from Eqs.lO and JEjl 
we finally obtain the TAP equation as 

(xi) N =F\ ^2 Jijk(xj)N{xk)N - r S csNA(a;i)jv)(31) 

with Tscsna and the form of the transfer function F 
self-consistently determined by the above Eqs.JSSJ), (|26ll . 
(|77jl . lO and (J201I. Noting that Tscsna can be ex- 
pressed in terms of Edwards- Anderson order parameter 
q, the TAP equation l|31[l can be further rewritten as 
(xi) N = F{Y JJ<k{ ^ l) Jt ] k{x J ) N {xk)N-af3q{q-q){x l ) N ), 
where the Edward-Anderson order parameter q and q 
are defined respectively as q = -k Yli { x i)n 2 and q = 
jj J2j ( x i) n which are easily computed using Eqs.@ and 

to. 

In conclusion, taking advantage of the close relation- 
ship between the TAP equation and the SCSNA ap- 



proaches, we have studied the SCSNA framework and the 
TAP equation for the stochastic analog network with 3- 
body interaction based on the Hebb learning rule. Char- 
acteristic to the TAP equation for the case of such in- 
teraction is that the Onsager reaction term consists of 
the two terms, as is shown in Ea. (|17fl : the one arising 
from the variance of the Gaussian distribution for the lo- 
cal fields that determines the shape of the transfer func- 
tion F and the other one due to the two body correla- 
tion of soft spins the computation of which requires tak- 
ing two cavities. Without resorting to the Hamiltonian 
based on overlap evaluation by adding a memory pattern 
to the network, which is usually taken for the Hopfield 
model @>E3, the coefficient of the Onsager reaction term 
has been given within the framework of the SCSNA, al- 
though applying the conventional recipe recovers the re- 
sult. Renormalization of the noise part of the local field 
in the SCSNA scheme corresponds to the overlap evalua- 
tion in the pattern-adding approach of the cavity method. 
The framework of the SCSNA described in its application 
to the (pre-)TAP equation of our stochastic network im- 
plies that the set of order parameter equations obtained 
there still formally makes sense in the case of general 
deterministic analog networks having 3-body interaction 
and transfer function F of arbitrary shape irrespective of 
whether it is monotonic or not. Details of the analysis 
including the phase diagram showing the behavior of the 
storage capacity will be published elsewhere. 
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